Mini Project: Training an MLP on MNIST

Deep learning is not well-understood, and the practice is ahead of the theory in many cases. If you are new to deep learning, you are strongly encouraged to experiment with many models, to develop intuition about why models work. Starter code is provided on github.

In this mini project, you'll modify the neural network in mnist_mlp.ipynb and compare the resultant model configurations.

Remember: Overfitting is detected by comparing the validation loss to the training loss. If the training loss is much lower than the validation loss, then the model might be overfitting.

Instructions

Task Description:

Take note of the validation loss and test accuracy for the model resulting from each of the below changes. (Try out each amendment in a separate model.)

Task List:

Increase (or decrease) the number of nodes in each of the hidden layers. Do you notice evidence of overfitting (or underfitting)?

Increase (or decrease) the number of hidden layers. Do you notice evidence of overfitting (or underfitting)?

Remove the dropout layers in the network. Do you notice evidence of overfitting?

Remove the ReLU activation functions. Does the test accuracy decrease?

Remove the image pre-processing step with dividing every pixel by 255. Does the accuracy decrease?

Try a different optimizer, such as stochastic gradient descent.

Increase (or decrease) the batch size.

Task Feedback:

Great job! Now, you're ready to learn about Convolutional Neural Networks (CNNs)!

Optional Resource

If you're interested in learning how to do a more systematic hyperparameter search in your neural networks, please check out this blog post.